Question/Answer (QA) systems may engage Subject Matter Experts (SMEs) to create the ground truth (candidate QA pairs) as part of QA system training. Generally, the process for this may have a unique set of questions that are given to each SME to create this set. QA systems, as well as other non-QA systems, may also employ user feedback as part of the QA system training. For example, some systems may provide a user interface with a binary classification option to the user requesting them to proactively flag an answer with some indication of whether the answer received is relevant to the question posed, and may also provide the user with the means to leave behind a comment. Thus, current QA systems typically rely on explicit user input and/or feedback to determine the effectiveness of an answer that is generated in response to a search query.